Optimizing Query Processing in Batch Streaming System

With the growing need of processing “big data” in real time, modern streaming processing systems should be able to operate at the cloud scale. This imposes challenges to building large scale stream processing systems. First, processing tasks should be efficiently distributed to worker nodes with small overhead. Second, streaming data processing should be highly available, despite that failures are common in datacenters. In Spark Streaming [26], the DStream model is proposed to cope the problems aforementioned. DStream stands for discretized stream; data in the incoming stream is divided into small batches for processing. Compared with processing data at the granularity of a record, batch processing has much lower overhead and has a cheaper fault tolerance model. Lineage information of each batch is kept for recomputation when failure occurs. Therefore, fault tolerance can be achieved without duplicating processing nodes. In this paper, we discuss how to optimize query processing in the DStream model. Specifically, we consider the case of Structured Query Language (SQL). SQL provides a declarative interface for the users query on the data. The declarative nature of SQL provides opportunity for query optimization as the execution is decoupled from the semantics of the query. In a streaming system, the same query is executed on similar data over and over again. Hence, the statistics of the data could be obtained for free, as long as the incoming data pattern is not changing abruptly. We study the performance of applying query optimization techniques in the DStream model, and show the advantage of dynamically optimizing stream processing.